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Maximum entropy approach to power-law distributions in coupled 

dynamic-stochastic systems. 
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Statistical properties of coupled dynamic-stochastic systems are studied within a com- 
bination of the maximum information principle and the superstatistical approach. The 
conditions at which the Shannon entropy functional leads to a power-law statistics are 
investigated. It is demonstrated that, from a quite general point of view, the power-law 
dependencies may appear as a consequence of "global" constraints restricting both the 
dynamic phase space and the stochastic fluctuations. As a result, at sufBciently long 
observation times the dynamic counterpart is driven into a non-equilibrium steady state 
whose deviation from the usual exponential statistics is given by the distance from the 
conventional equilibrium. 
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I. INTRODUCTION 



Power-law distributions are quite conimon for 
complex systems of different nature [1]. Several 
theoretical schemes have been developed in or- 
der to understand this behavior. The concepts 
of the self-organized criticality [2] and highly op- 
timized tolerance [3] have been extensively il- 
lustrated within various kinetic (sandpile, slider- 
block, forest fire, etc.) models [3,4], which exhibit a 
triggering between different regimes, accompanied 
by power-law dependencies - avalanches. More 
recent studies reported on similar phenomena in 
inelastic dissipative gases [5], stochastic processes 
with multiplicative noise [6] , clustering models [7] , 
and condensation in porous media [8]. Observa- 
tions of the power-low features in a variety of much 
more complex (physical, biological, social, etc.) 
systems [4,7] led to further searches for a generic 
mechanism, responsible for such a behavior inde- 
pendently of a system microscopic specificities or 
model approximations. 

A remarkable step was to find [9] that a power- 
law could appear from the usual exponential with 
fluctuating parameters. This is a basic idea of the 
superstatistical approach [10-12] explaining the 
power-law statistics in a system as a result of fluc- 
tuations in its surrounding (background) . The lat- 
ter can be modeled as a stochastic [9-13] or a dy- 
namic (Hamiltonian) [14,15] process. Nevertheless 
the information on the source of the background 
fluctuations is not always available. Moreover, in 
many cases (e.g. optimal control [16] or inser- 
tion into flexible environments [17]) the two coun- 
terparts are strongly coupled and the background 
statistics is (or should be) conditional to the sys- 
tem properties. In that case the background evolu- 
tion cannot be modelled as a process (e.g. fluctu- 
ating temperature [10] or mass [13]) independent 
of the system state. Lacking any detailed infor- 
mation on the coupling, one has to resort to the 



inference methods. 

The maximum entropy inference methods [18] 
have been developed in this context. These are 
based on parameterized information entropy mea- 
sures (Tsallis [19] or Renyi [20,21]) which have 
the Shannon form as a limit. Despite of various 
successful applications, this approach has gener- 
ated several controversial points [22] related to the 
meaning of the entropic index, the non-additivity 
effects, spurious correlations and the biased aver- 
aging [23]. This stimulated further steps [24,25] 
towards deriving non-exponential distributions by 
maximizing the Shannon entropy under suitable 
constraints. 

Therefore, it seems promising to develop a 
scheme, capable of merging the advantages of 
the maximum entropy and the superstatistical ap- 
proaches. More specifically, our main goal is to find 
out the conditions at which the Shannon entropic 
form leads to the background statistics, coherent 
with the one introduced within the superstatisti- 
cal approach. This would allow us to clarify the 
mechanism of the corresponding power-law depen- 
dencies and to understand the essence of the non- 
additivity effects. 



II. COUPLED SYSTEMS 

With this purpose we consider a many-body dy- 
namic (deterministic) system in contact with a 
fluctuating background. In contrast to coupled 
dynamic systems [26], in our case the two subsys- 
tems are of different origin and thus require differ- 
ent levels of description [27]. The background (the 
stochastic system) is considered to be a source of 
some relevant quantity, f3 (e. g. pore size, temper- 
ature, etc), which fluctuates according to a prob- 
ability distribution /(/3). Quite often the differ- 
ent nature of the two subsystems leads to a well- 
defined separation of the relaxation time scales 
[12,13], typical for self-organizing or glassy sys- 



terns. Namely, the system relaxation is supposed 
to be much faster than the background fluctua- 
tions. Such that for any background state the sys- 
tem can reach an equilibrium (or stationary) state 
with a conditional thermodynamic function 9{p\P). 
In principle, this could be any function, suitable 
for an adequate description of the system internal 
order and linking the relevant extensive parame- 
ters p to the intensive ones (3. In what follows the 
function 0{p\(3) is assumed to be known (from ex- 
act results or relevant approximations). Then an 
observable can be represented as an average over 
the background fluctuations 



Tip) = e{p\f3) 



dl3f{(3)e{p\l3) 



(1) 



where the overbar denotes the corresponding av- 
eraging. Note that the time scales separation is 
essential, otherwise the "quenched" average in (1) 
docs not make sense. 

If the background does not undergo some inter- 
nal stochastic process independently of the system, 
then ,f{p) is a priori unknown. It should be deter- 
mined from the information on T{p). This prob- 
lem is typical for characterizing the heterogeneous 
media through indirect (e.g. adsorption) probes, 
where T{p) is a measurement result. On the other 
hand, in many applications (e.g. protecting stor- 
age, adaptive learning and control [16,25]) it is de- 
sirable to design the background in the way leading 
to a well-defined behavior T{p). In this case T{p) 
should be considered as a cost function. 



III. MAXIMUM ENTROPY APPROACH 

Therefore, one deals with an inverse problem 
of extracting /(/3) from T{p). This can be done 
within the maximum-entropy inference scheme 
proposed by Jaynes [18]. Our uncertainty on the 
background state can be estimated by an informa- 
tion entropy, which is taken in the Shannon form. 



H = - /d/3/(/3)ln[/(/3)] 



(2) 



Maximizing H under the constraint (1) and requir- 
ing the normalization for /(/3) we get the following 
conditional distribution /(/3) = ,f{l3\p) [17] 



fiPlp) 



-«e(p|/3) 



Z= d(3e 



-Ke(p\[j) 



(3) 



where the Lagrange multiplier k should be found 
from the constraint (1), that is equivalent to solv- 
ing 

T(p) = -AlnZ 



Plugging the distribution (3) back to (2) we obtain 
the amount of information (on the background) 
one can get by driving (e.g. through varying p) 
the dynamic subsystem 



H{p) = KT{p) + \nZ 



(4) 



In particular, the information rate takes a remark- 
ably simple form 



dH{p) 
dp 



dT{p) de{p\p) 



dp 



dp 



(5) 



It is clear that the scheme gives a solution for /(/?) 
which is free from adjustable parameters, provid- 
ing an explicit link between the data (or cost func- 
tion) and the conditional theoretical estimation. 
Therefore, the sensitivity to the " kernel" variation 
9{p\l3) and to scattered data T{p) can be easily 
controlled. On the other hand, if /(/3) is already 
known, then our results can be used to estimate 
the quality of the model 0{p\(3) through its match- 
ing to the available data T{p). 



IV. CONSTRAINTS 

Despite of the apparently exponential form (3) 
the actual behavior of /(/3) depends on the na- 
ture of the constraint imposed and on a form of 
the constrained function 0{p\j3). Such an ambigu- 
ity should not be considered as a shortcoming of 
the theory. This is a consequence of the fact that 
we are working under conditions of incomplete in- 
formation. Then, according to the Baesian inter- 
pretation, a probability should be considered as a 
measure of our ignorance rather than an objective 
property. Nevertheless, our freedom in choosing 
the constraints is restricted by any prior informa- 
tion, coming through independent tests. On the 
other hand, there are constraints which are "natu- 
rally" imposed either as design principles [28] or as 
experimental conditions. In what follows we dis- 
cuss two relevant examples. 



A. Entropy constraint 

As is mentioned earlier, in many applications it 
is desirable to constraint the system internal or- 
der with the purpose of meeting some survival or 
functionality objectives. In this case it is nat- 
ural to restrict the phase space by constraining 
the thermodynamic entropy S{p\(i). This idea 
was shortly discussed in a slightly different con- 
text [17,25]. Therefore, we set e{p\(3) = S{p\f3), 



T{p) = S(p). Then the distribution (3) closely 
resembles the Einstein fluctuation formula. Note 
however that /(/3) describes the background fluc- 
tuations and for k = 1 it becomes identical to the 
distribution of the system fluctuations. In partic- 
ular, for small fluctuations around an equilibrium 
state (p, P*) we may expand 



S{p\P) = S{p\p* 



1 



2x(p,/3*) 



iP-n' 



(6) 



where xiPiP*) — {{P — P*Y) is the mean-square 
fluctuation in the system when the background 
state is fixed aX [3 = j3* . In this approximation 
the distribution (3) becomes gaussian and k can 
be determined combining (6) and (1). Finally we 
arrive at 



{p-p*Y^{{p-p*Y)[s{p\n-np)] (7) 

Note that S{p\l3*) is the system equilibrium cn- 
tropy C onsequently S{p\(3*) - i;(p) > and 
{(3 — I3*Y > as it should be. Therefore, in order 
to ensure a given response, S(p), the background 
should fluctuate coherently with the system fluctu- 
ations and with the distance from the equilibrium 
state. As we will see below, for large fluctuations 
this tendency also takes place. A quite similar 
trend has been reported [23] for fluctuations in the 
Tsallis statistics. In the limit of S{p\(3*) = S(p) we 
return to the standard equilibrium without fluctu- 
ations in the background: f{f3) = S{P - /3*). The 
system fluctuates according to its response func- 
tion x(/0,/3*)- 

In order to study large background fluctuations 
and the system statistics at different time scales we 
have to introduce an explicit form for S{p\/3). With 
this purpose we consider an exactly solvable toy 
model - the ideal gas in contact with a reservoir of 
fluctuating temperature [10,11]. This choice is mo- 
tivated by our goal to extract the most general and 
essential features, independent of approximations 
or the system correlations. Therefore, we deal with 
the entropy per particle S{p\f3) = 5/2 — ln{pA^). 
Here p is the number density, /? ~ 1/kT is the in- 
verse temperature and A is the thermal de Broglie 
length. Introducing irrelevant scaling constants 
(making p and (3 dimensionless), S{p\P) can be re- 
duced to 

S{p\f]) = const - ln(p) - ^ ln(/3) (8) 

Constraining the average temperature 



(9) 



/3o = / d(3fiP)(3 
and the entropy 



S(p) - / df3f{(3)S{p\p) 



(10) 



through the inference procedure discussed above, 
we obtain the following distribution 



/(/?) = 



pK^-l3/f3{K) 



(«;)]''+ir(K+l)' 



/?(«) = 



/3o 



(11) 



which is precisely the F-distribution considered in 
the supcrstatistical approach [10,11], where the ex- 
ponent K is related to the noise intensity. In our 
case the Lagrange multiplier k should be deter- 
mined from the entropy constraint (10) 

E(p)=5(p|/3o)+ln(«: + l) -*(«:)-! (12) 

K 

where ^(k) =: dlnT{K)/dK. Thus, the exponent 
K is determined by the distance S(p) — S{p\f3o) 
from the equilibrium state {p,f3o). In particular, 
for large k we have found 

S(p) - S{p\(3o) + 1/(2«) 

Therefore, k is related to the deviation from the 
standard equilibrium, such that f{(3) — > S{P — /3o) 
and S(p) = S{p\Po) as k ^ oo. 

At the short-time scale (of the order of the sys- 
tem relaxation time) the conditional energy distri- 
bution (E = p^ /2m, at a given temperature (3 ) is 
Gibbsian 



f{E\l3) = 



-!3E 



JdEe- 



■I3E 



(13) 



The long-time behavior of the dynamic system can 
be represented as a superposition of its short-time 
statistics and the background fluctuations - the 
supcrstatistical approach [10,11]. In this spirit the 
long-time energy distribution can be found by av- 
eraging over the temperature fluctuations 



f{E) = / dpf{[3)J{E\(3) - /3o 



(3oE_ 
1-9 



(14) 



where q = k + 2. For E = p^ /2m we recover the 
power-law velocity distribution found [10,11] in the 
superstatisical approach. Quite similar effects have 
recently been predicted to occur in driven dissi- 
pative inelastic gases [5] and in driven stochastic 
systems with multiplicative noise [6] or fluctuat- 
ing mass [13] In a different context similar power 
laws where found, applying the maximum-entropy 
inference to parameterized entropies (Tsallis [19], 
and Renyi [21]). But the meaning of the entropic 
parameter is not always clear, while in our case q 
is directly related to the constraint imposed. 



Thus, because of the constraint, imposed on the 
internal order at longer times the system devel- 
ops avalanches (or energy cascades) at any finite 
K. The avalanche size is characterized by 1/k. 
Therefore, maintaining the distance from the equi- 
librium, one can control the size of the rare "catas- 
trophic" events. This can be organized in different 
ways, such as by powerful energy injections at large 
velocity scales [5] , or through an interplay of addi- 
tive and multiplicative noises [6]. 



B. Activity constraint 

Other constraints are realized as driving condi- 
tions. For instance, adsorption into porous media 
[8] is driven by a difference between the chemi- 
cal potential in the fluid bulk, /if, and that inside 
the matrix M{f3, p). Adsorption equilibrium corre- 
sponds to M(/3, p) — pb- Nevertheless, recent anal- 
ysis [8] of experimental results reveals that the true 
equilibrium seems to be hardly reachable because 
of very long equilibration times and well-developed 
metastability. At certain conditions this leads to 
the hysteretic behavior accompanied by multiple 
metastable states of the fluid. On the other hand, 
a complicated matrix topography makes one to re- 
sort to a statistical description (e.g. pore sizes, 
site energies, etc.). In this context the matrix can 
be considered as medium inducing a distribution 
/(pi.../?jv) of metastable states with local fluid den- 
sities Pi in different spatial domains i = 1..N. For 
simplicity the domains are assumed to be uncorre- 
lated: f{pi---PN) =Yli f{pi)- If the temperature /? 
is fixed, then the fluid state in each domain is given 
by a local isotherm p{/3\pi). The overall isotherm 
is an average over the domains 



M{p,p)^ j dp,f{pi)p{p\p,) (15) 

where p is the average fluid density given by 



dpif{pi)pi 



(16) 



The local isotherm is chosen in the ideal gas form 
Pp(f3\pi) = hi(piA^). Maximizing the Shannon en- 
tropy under constraints (15), (16) we obtain the 
following local density distribution 



f{P^ 



[p,/p{X)]\.-p^/p(^^ 

p(A)r(i + A) • 



pW 



p 



l + A 



(17) 



which is (for — 1 < A < 0) of the form taken in 
[8] as a fitting function for the description of the 
collective condensation events (avalanches). The 
Lagrange multiplier A should be determined from 
the constraint (15) 



M(/3, p) = p{(i\p, = p)- ln(A + 1) + vI/(A) + - 

(18) 

Therefore, the exponent A is again related to the 
distance M{(3,p) — p{j3\pi — p) from the conven- 
tional equilibrium. The latter appears as a limit 
of A ^ oo, leading to f{pi) — S{pi — p) when 
M{(i,p)=p{(i\p,=p). 

It should be noted that the driving procedure is 
extremely important. If, for instance, the chemical 
potential (pressure) is free to relax according to a 
controlled particles injection, then the constraint 
(15) should be removed (A = 0) and the system 
follows a different path with a purely exponential 
distribution. This agrees with the conclusion made 
in [8]. Moreover, it can be easily shown that a de- 
pendence on the driving path is a generic feature 
of the coupled systems considered here. For this 
purpose let us consider an adsorption of noninter- 
acting species into a network with fluctuating site 
binding energy e, distributed according to some 
probability density /(e). If the process is driven 
by increments in the chemical potential /x, then 
the conditional grand potential is 

(19) 



n(/z|e) = -ln(l 



oM+e\ 



The adsorption isotherm (coverage T vs p) is given 

by 



Tip) 



dn{p\e) 



eP4 



(20) 



dp \1 + e^+^ ^ 

On the other hand, if the coverage is maintained 
by controlled injections, then the conditional free 
energy is 

F(e\€) = -€e + 6\n6+{l-e) ln(l - 61) (21) 

This allows to calculate the chemical potential 



P- 



dF{e\e) 



= -e + In ■ 



de — -~ ^ 

which can be formally solved with respect to 6 



(22) 



e = — — — (23) 

Therefore, the isotherm strongly depends on the 
driving conditions. The main reason is a non-zero 
distribution width (fluctuations) in the stochastic 
counterpart. This can be demonstrated by expand- 
ing (20), (23) in terms of e and analyzing the dif- 
ference 



T -6 = 



n=2 



a„(M)F-e"] 



(24) 



which vanishes only if the distribution is (5-like (no 
fluctuations). Note that our conclusion is quite 
general. It does not depend on a shape of the dis- 
tribution /(e) and on the adsorbate dynamics. 



V. CONCLUSION 

We have found that the major ingredients, rel- 
evant to the power-law distributions in composite 
systems are (i) the widely separated times scales, 
(ii) non-vanishing background fluctuations, (iii) a 
constraint, imposed on the overall system, hold- 
ing the dynamic counterpart in a stationary non- 
equilibrium state. 

One might argue that the T- and the power-law 
distributions result trivially from the logarithmic 
form of the constrained functions. It should be 
noted, in this respect, that the logarithmic shape 
of the thermal entropy is of quite general nature, 
as this follows from the famous Boltzmann relation 
5* — k\nW. The density distribution (17) is also 
not a specific feature of the ideal gas. Our results 
do not alter if the interparticulc interactions are 
taken into account (e.g. as a long-range perturba- 
tion [29]): 

/3M'(/3|p^)-ln(p^A3)-[/pi 

In that case the distribution (17) does not change 
and the exponent A is determined by the distance 
from the equilibrium state iJ,'{f3\pi — p). 

The main advantage of the approach developed 
here is that it avoids parameterized entropy mea- 
sures and allows one to apply the superstatistical 
scheme to coupled systems in which the stochas- 
tic background does not fluctuate independently of 
the dynamic counterpart. 

In the context of our study the nonextcnsitivy 
in systems with power-law distributions is a di- 
rect consequence of the "global" nature of the con- 
straint. This makes it impossible to decompose the 
system into non-correlated parts. Moreover, for 
any background distribution of non-zero width, the 
system thermodynamic response strongly depends 
on a driving path. 
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